Experimentation at Twitter

Why experimentation?

• Most of the business questions are causal in nature
  • If we change X, how will it improve Y?
• X might be a feature we want to introduce, Y might be DAU
• Answering these questions are tricky and error-prone (bias everywhere)
• Experimentation is the Gold Standard!

A/B testing in Obama’s Presidential Campaign

Variation 1

Variation 2

Monumental Successes!

Overview

Experimentation is at the heart of Twitter’s product development cycle:

• Hundreds of experiments each quarter
• Very typical to have millions of users in each bucket
• Test from UI, front-end changes to algorithmic, back-end changes
• A collective efforts of PM, DS, and engineering

Changing the Favorite Icon From Start to Heart

Improve the Relevance of Who to Follow Algorithm

Adding a Recap Module to the Home Timeline

How to run an experiment?

Gather Samples

• Who are eligible for the experiments?
• How many samples do you need for each bucket?
• How long do you need to run the experiment for?
• Learn about Statistical Power, False Positive, and Effect Size

Bucket Size Estimator

Assign Buckets

• Ensure that bucket assignment is truly randomized (Use a proper Hash function)
• Do so by sanity check metrics that should be not different across buckets
• The number of users being bucketed should be roughly the same (Bucket Health Table)

Bucket Health

Apply Treatment

• Dealing with hundreds of simultaneous experiments & potential collision
• Experiment Groups - Shared/Splitted experiment groups

Measure Outcome

• Always Have a hypothesis in mind
• Define your success/failure metrics ahead of time
• Understand the trade-offs you are willing to tolerate to begin with

Adding Track Events

Make Comparison

• How to know the observed difference is a signal, not noise?
• Statistal Tests, P-values, Confidence Intervals
• Be aware of False Positives. Do not peak at experimental Results

The probelm of early peaking

Limitation of A/B Testing

• No understanding of mechanism
• Emphasis on short-term results
• Novelty Effect
• Implementation Costs
• Deliver inconsistent UX
• OEC may be hard to define
• Not everything can be tested

Thank you!

• Please follow me on Twitter at @_rchang! or shoot me an email at rchang@twitter.com
• Happy Experimenting!